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Abstract 

We suggest a new perspective of research towards understanding the relations be- 
tween structure and dynamics of a complex network: Can we design a network, e.g. 
by modifying the features of units or interactions, such that it exhibits a desired 
dynamics? Here we present a case study where we positively answer this question 
analytically for networks of spiking neural oscillators. First, we present a method 
of finding the set of all networks (defined by all mutual coupling strengths) that 
exhibit an arbitrary given periodic pattern of spikes as an invariant solution. In such 
a pattern all spike times of all the neurons are exactly predefined. The method is 
very general as it covers networks of different types of neurons, excitatory and in- 
hibitory couplings, interaction delays that may be heterogeneously distributed, and 
arbitrary network connectivities. Second, we show how to design networks if further 
restrictions are imposed, for instance by predefining the detailed network connectiv- 
ity. We illustrate the applicability of the method by examples of Erdos-Renyi and 
power-law random networks. Third, the method can be used to design networks that 
optimize network properties. To illustrate the idea, we design networks that exhibit 
a predefined pattern dynamics and at the same time minimize the networks' wiring 
costs. 
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1 How does network structure relate to dynamics? 

Our understanding of complex systems, in particular biological ones, ever more 
relies on mathematical insights resulting from modeling. Modeling a complex 
system, however, is a highly non-trivial task, given that many factors such as 
strong heterogeneities, interaction delays, or hierarchical structure often occur 
simultaneously and thus complicate mathematical analysis. 

Many such systems consist of a large number of units that are at least qual- 
itatively similar. These units typically interact on a network of complicated 
connectivity. Important example systems range from gene regulatory networks 
in the cell and networks of neurons in the brain to food webs of species being 
predator or prey to certain other species P)17jll| . 

A major question is how the connectivity structure of a network relates to 
its dynamics and its functional properties. Researchers therefore are currently 
trying to understand which kinds of dynamics result from specific network 
connectivities such as lattices and random networks as well as networks with 
small-world topology or power-law degree distribution. [3T]I26II30| 

Here we suggest a complementary approach: network design. Can we modify 
structural features of a complex network such that it exhibits a desired dy- 
namics? We positively answer this question analytically for a class of spiking 
neural network models and illustrate our findings by numerical examples. 

In neurophysiological experiments, recurring patterns of temporally precise 
and spatially distributed spiking dynamics have been observed in different neu- 
ronal systems in vivo and in vitro [T9|29ll37lfT4] . These spike patterns correlate 
with external stimuli (events) and are thus considered key features of neural 
information processing |2j. Their dynamical origin, however, is unknown. One 
possible explanation for their occurrence is the existence of excitatorily cou- 
pled feed-forward structures, synfire chains [T|15|10|5] . which are embedded 
in a network of otherwise random connectivity and receive a large number 
of random external inputs. Such stochastic models explain the recurrence of 
coordinated spikes but do not account for the specific relative spike times of 
individual neurons, although these are discussed to be essential for computa- 
tion, too. To reveal mechanisms underlying specific spike patterns and their 
computational capabilities, our long term aim is to develop and analyze a 
new, deterministic network model that explains the occurrence of specific pre- 
cisely timed spike patterns exhibiting realistic features. The work presented 
here constitutes one of the first steps in this direction (cf. also [T8|20f9] ) and 
focuses on designing networks such that they exhibit an arbitrary predefined 
periodic spike pattern. 

The article is organized as follows. In section 2 we introduce a class of network 
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models of spiking neurons and illustrate their relation to standard modeling 
approaches using differential equations. In section 3 we design networks by de- 
riving systems of equations and inequalities that analytically restrict the set 
of networks (in the space of all coupling strengths) such that they exhibit an 
arbitrary predefined periodic spike pattern as an invariant dynamics. It turns 
out that such systems are often underdetermined such that further require- 
ments on the individual units, the interactions and the network connectivity 
can be imposed. We illustrate this in section 4 by specifying completely, for 
each neuron, the sets of other neurons it receives spikes from, i.e. the entire 
network connectivity. We present examples of networks with specified connec- 
tivities of different statistics and design their coupling strengths such that they 
exhibit the same spike pattern. In section 5 we demonstrate the possibility of 
designing networks that are optimal (with respect to some cost function) . We 
present illustrating examples of networks that exhibit a certain pattern of pre- 
cisely timed spikes and at the same time minimize wiring costs. In section 6 
we provide a brief step-by-step instruction for applying the presented method. 
Section 7 provides the conclusions and highlights open questions regarding the 
design of complex networks. 

The method of finding the set of networks exhibiting a predefined pattern 
(parts of sections 2 and 3 of this article) was briefly reported before in refer- 
ence [23] and in abstract form in [22], where only the case of non-degenerate 
patterns, identical delays and identical neurons was treated explicitely. Small 
inhomogeneities have been discussed in [9]. Here we include also degenerate 
patterns, heterogeneously distributed delays and allow for different neuron 
types. Moreover, we present new applications of network design, see in partic- 
ular sections 4 and 5. 



2 Model neural networks 

2.1 Phase model 

Consider a network of oscillatory neurons that interact by sending and 
receiving spikes via directed connections. The network connectivity is arbitrary 
and deflned if we specify for each neuron / G {!,..., N} the sets Pre(/) from 
which it receives input connections. One phase-like variable 0;(t) specifies the 
state of each neuron / at time t. A continuous strictly monotonic increasing rise 
function Ui, Ui{0) = 0, defines the membrane potential Ui[(f)i) of the neuron, 
representing its subthreshold dynamics [25], see Fig. [H The neurons interact 
at discrete event times when they send or receive spikes. We first introduce 
the model for non-degenerate events, i.e. non-simultaneous event times, and 
provide additional conventions for degenerate events in the next subsection. 
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Figure 1. Phase dynamics in response to incoming excitatory spike. The rise func- 
tion Um of neuron m is plotted as a function of its phase cpm- In the absence of 
interactions, (f)rn{t) increases uniformly with time t according to Eq. ([T]). If a spike 
is sent by nGuron / ctt tiniG it is rGCcivGcl by nGuron tti ctt tiniG t -\- and induces 
a phase jump (j)m{it + Tmi)~) — > 4>mit + Tmi) that is mediated by the rise function 
Um and its inverse according to ^ and Here Qu,m = Um{&m) is the threshold 
for the membrane potential, cf. sec. 12.31 

In the absence of interactions, the phases increase uniformly obeying 

d(f)i/dt = 1. (1) 

When (pi reaches the (phase-) threshold of neuron /, 4>i{t^) = Qi > 0, it is 
reset, 0;(t) = 0, and a spike is emitted. After a delay time Tmi this spike signal 
reaches the post-synaptic neuron m, inducing an instantaneous phase jump 

<Pm {t + Tml) = H^"2] (it + Tml)-)) , (2) 

mediated by the continuous response function 

Hi"^\<p) = u;^\Um{<P) + s) (3) 

that is strictly monotonic increasing, both as a function of e and of 0. Here, 
Smi denotes the strength of the coupling from neuron / to m. This coupling is 
called inhibitory if Smi < and excitatory if Smi > 0. We note that sending and 
receiving of spikes are the only nonlinear events occurring in these systems. 
Throughout the manuscript, 4>i{t) is assumed to be piecewise linear for all / 
such that in any finite time interval there are only a finite number of spike 
times. 
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2.2 Degenerate event timing 



These events of sending and receiving spikes might sometimes occur simulta- 
neously such that care has to be taken in the definition of the model dynamics. 
Simultaneous events occurring at different neurons do not cause any difficulties 
because an arbitrary order of processing does not affect the collective dynam- 
ics at any future time. However, if two or more events occur simultaneously at 
the same neuron, we need to specify a convention for the order of processing. 
We will therefore go through the possible combinations in the following: 

(i) spike sending due to spike reception: The action of a received spike might 
be strong enough such that the excitation is supra-threshold, 

U„, {(pm [{t + Tmiy)) + Sml > Um (Qm) ■ (4) 

We use the convention that neuron m sends a spike simultaneous to the re- 
ception of another spike from neuron / at time t + r^i and is reset to 



^m{t + T^l) = 0. (5) 

(ii) spike received at sending time: If neuron m receives a spike from neuron I 
exactly at the same time when m was about to send a spike anyway, 

0m {{t + T^iy^ = Qm, (6) 

we take the following convention for the order processing: first the spike is sent 
and the phase is reset to zero, then the spike is received such that 

0m it + r™0 = (0) . (7) 

If the spike received causes again a supra-threshold excitation, we neglect a 
second spike potentially generated at time t + Tmi and just reset the neuron 
m to zero as in (I5|). 

(iii) simultaneous reception of multiple spikes: If multiple spikes are received 
simultaneously by the same neuron and each subset of spikes does not cause a 
supra-threshold excitation (as in dH)), a convention about the order of treat- 
ment is not necessary as can be seen from the following argument. If neuron 
m at time 9 simultaneously receives G N spikes from neurons li,...,lh, and 
a : {l,...,h} — > {l,...,h} is an arbitrary permutation of the first h integers, 
we have 
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= f/„^[f4i(0m(6'")) + ^m/^f;,) + ••• + £^m«,(2) + 



Treating the incoming spikes separately in arbitrary order is therefore equiv- 
alent to treating them as one spike from a hypothetic neuron with coupling 
strength Smh ^^mh + ••• ~^^mih to neuron m. Moreover, upon sufficiently small 
changes of the spike reception times, the sub-threshold response of a neuron m 
continuously changes with these reception times, even if their order changes: 
For every ordering of the reception times, the total phase response converges, 
in the limit of identical times, to the phase response to simultaneously re- 
ceived spikes. This is because the neuron's response function H^"^^ is identical 
for different incoming spikes. We note that this might not be the case in neu- 
robiologically more realistic models if they take into account that spikes from 
different neurons arrive at differently located synapses. These spikes may have 
a different effect on the postsynaptic neuron even if they generate the same 
amount of charge flowing into (or out of) the neuron. 

We extend the deflnition 

MO) = Htl_,,^,^^,,,_,,^,^{Mo-)) (9) 

for the processing of multiple spike receptions to more involved cases, where 
a subset of spikes generates a spike. Treating this subset first would result 
in a different dynamics than summing up all couplings strength, e.g. if the 
remaining couplings balance the strong excitatory subset. In this case the order 
of treatment is not arbitrary and the phase as well as the spikes generated in 
response to the receptions do not continuously depend on the spike reception 
times; as a convention, we sum the coupling strengths first, as in ((91). 

The generalization of (i) and (ii) to the case of multiple spikes received simul- 
taneously is straightforward. The dynamics however will in general also not 
depend continuously on the reception times. 

(iv) simultaneous sending of multiple spikes: As we exclude the simultaneous 
sending of multiple spikes by the same neuron, if several spikes are sent si- 
multaneously, they are sent by different neurons; therefore no difficulties arise 
and we need no extra convention. 
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2.3 Phases vs. neural membrane potentials 



The above phase dynamics in particular represent (cf. also [25|12II32II35|36| ) 
dynamics of neural membrane potentials defined by a hybrid dynamical sys- 
tem [4j consisting of maps that occur at discrete event times and ordinary 
differential equations, or, formally, of a differential equation of the form 

^ = fm{Vm)+Ut). (10) 

Here Im(t) = J2i,n^miS{t — ti^n — Tmi) is a sum of delayed ^-currents induced by 
the neurons / G Pre(m) sending their nth spike at time ti^n- A solution Vm{t) 
gives the membrane potential of neuron m at time t in response to the current 
from the network Im(t). See Fig.[2]for an illustration. A spike is sent by neuron 
m whenever a potential threshold is crossed (for supra-threshold input, e.g., 
^m(C,n) + ^mi > Qu,m fov somc l] othcrwisc Vm{t:;n,n) = 0c/,m), leading to 
an instantaneous reset of that neuron, Vm{tm,n) = (or to a nonzero value 
equal to the coupling strength of the incoming pulse, if a subthreshold spike 
reception coincides with the potential satisfying Kn(t^ n) — ©c/.m, according to 
(ii) in sub-section 12.21) . The positive function fmiy) > (for all admissible V) 
yields a solution Vm(t) of the free (Im = 0) dynamics that satisfies the initial 
condition Vm{0) = 0. We continue this solution Vm on the real interval t G 
{B, 9m], i.e. to negative real arguments t with infimum B G M~U{— cx)} and to 
positive real t until 0^ G M"*" where Kn(0m) = ©i/.m • We note that a too large 
inhibition can be inconsistent with a possible lower bound \im^\BVm(y(p) > 
— oo of the membrane potential as present, e.g., for the leaky-integrate-and- 
fire neuron with 7 < (cf. Eq. (flGll). However, it does not change the methods 
developed below using the phase representation and is therefore not considered 
in the following. The above rise function Um is then defined via Vm as 

UU<P) := V„(0), (11) 

where (j) E {B, Qm]- The potential dynamics can now be expressed in terms of 
a natural phase (pmit) such that 

Vm{t) = UmiKit)) (12) 

for all t. Since Vmif) is strictly monotonically increasing in t, this also holds for 
Umi4>) in 01 cind the inverse exists on the interval (lim^^^ ym(0), 0(7,m]- 
Therefore, the phase at the initial time, say to, can be computed from the 
initial membrane potential via 0m(^o) = U~^{Vm{to)). If the dynamics evolves 
freely, the phase satisfies d(j)rn/dt = 1, and is reset to zero when its threshold 
9m is reached, cf. Fig. [2l Due to the invertibility of Um, there is a one-to-one 
mapping 
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Figure 2. Relation between phase and membrane potential dynamics. (a,b) Dynamics 
of membrane potential Vm{t) of neuron m. (a) The free dynamics is periodic with 
period Tg^m; (b) dynamics in response to an incoming excitatory spike at time 6. 
(c,d) Dynamics of (pm{t) representing a phase-like variable of the membrane potential 
dynamics displayed in panels (a) and (b). (c) Periodic phase dynamics has the same 
period To^rn', (d) dynamics in response to input implies phase jump given by Eq. jj]). 



= U^' {Qu,m) (13) 

between the threshold Qu,m in the membrane potential and the threshold 9^ 
in the phase. This phase threshold equals the free period of neuron m, 

em = To,m, (14) 

due to the constant unit velocity ([T]) of the phase in the absence of input: 
starting from zero after reset, the phase (pm needs a time 6^ to reach the 
threshold. Thus 9m is the intrinsic inter-spike-interval and l/6m is the intrin- 
sic frequency of neuron m. 

In the presence of interactions, the size of the discontinuities in the phase 
resulting from spike receptions have to match the size of the corresponding 
discontinuities in the membrane potential, cf. Figs. [T] and [2l To compute the 
correct size, we first compute the membrane potential Vm{9~) = Um{(pm{0~)) 
of neuron m just before the reception time ^ of a spike from neuron /. The 
membrane potential after the interaction is given by Vm{0) = Um{4>m{0~)) + 
Smi due to (fTOl) . We return to the phase representation using the inverse rise 
function and compute the phase after the interaction 

<Pm{e) = f/m^(K„(0)) = f/„^(t/m(0m(r)) + £w) = i^t)(0m(^-)), (15) 
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and arrive at relation (I2|) between the phase before and after interaction. To- 
gether with the fact that the reset levels, the thresholds and the free dynamics 
match due to f/~^(0) = 0, Eqns. (fT3l) and (fTTl) . this shows the equivalence of 
the membrane potential dynamics given by the hybrid system ffTOll and the 
phase dynamics defined in section I2.1[ 

As an important example, the leaky integrate- and- fire neuron, defined by 
fmiy) = I — iV, results in the specific form 

t/iF(0) = (//7)(l-e-^^). (16) 

Here / > is a constant external input and 7 G M specifies the dissipation 
in the system. For normal dissipation, 7 > 0, ?7if(0) is concave, Ulp{(j)) < 0, 
bounded above by 1/7 and it approaches this value for ^ cxd. Assuming 
7/7 > Qu we obtain an intrinsically oscillatory neuron. For 7 < 0, Uip{(f)) is 
convex, U^p^cf)) > , and bounded below by 1/7 < 0. It grows exponentially 
with (j) such that, apart from Qu > 0, no condition is necessary to obtain 
a self-oscillatory neuron. For 7 = 0, the dynamics of an isolated neuron is 
trivial and specified by ?7if(0) = The phase-threshold (fT3ll for a particular 
integrate-and-fire neuron m is given by 

©m = U^'^{Qu,m) = 7m ln(/m/(/m " 7m0(7,m)) (17) 

if the parameters are and 7^; for 7^ = we have &m = 'du,m/Im, the limit 
7m ^0 in dlTl). 

Another interesting and analytically useful example is given by the biological 
oscillator model first introduced by Mirollo and Strogatz [25j, 

t/Ms(0) = rMn(l + a-V), (18) 

ab > 0, which result from a differential equation (fTOl) with 
fm(y) = exp{—bV)/{ab). Here Uusi'P) is concave for a,b > and convex 
for a,b < 0. In the former case the domain of Ums is G (—a, 00), with 
Ums{4>) cxd as — > cxd; in the latter case the domain is G {—00, \a\), where 
Ums{4>) — * cxd as y \a\. Therefore, in both cases, there are no additional 
conditions on Qu. The threshold for the phase of a particular neuron m is 
given by 

= f^m^(0(7,m) = am{ey:p{bmQu,m) - 1) (19) 
for parameters am,bm- 

We note a direct relation between neural oscillators of leaky integrate-and-fire 
and Mirollo-Strogatz type: the rise function of a Mirollo-Strogatz oscillator is 
the inverse of the rise function of a leaky integrate-and-fire neuron. For x in 
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Figure 3. (color) Spike pattern in a small network {N = 7). (a,b) Network of four 
leaky integrate-and-fire (green) and three Mirollo-Strogatz (blue) neurons in graph 
and matrix representation. The parameters of the leaky integrate-and-fire neurons 
are randomly chosen within 7^ G (0.5,1.5), Im = (1.08,2.08) and Qm S (0.5,1.5). 
(If 7m = 1 and Im = e/(e — 1) « 1.58 as well as @m = 1 then @u,m = 1-) 
The parameters bm of the Mirollo-Strogatz neurons are randomly chosen within 
bjn G (0.7, 1.5), then is chosen within G (l/(e^'" - 1) - 0.1, l/(e'''" -1) + 0.1) 
and @rn G (0.5,1.5). The delays are randomly distributed within Tmi G (0.1,0.9). 
Connections are either excitatory (black) or inhibitory (red) . In (a) the line widths 
of the links, in (b) the color intensities are proportional to the coupling strengths. 
The network is a realization randomly drawn from those networks with couplings 
in the range eim G (—1.5,1.5) that exhibit the predefined pattern displayed in (c) 
(black bars underlying the colored ones), (c) The spiking dynamics (green and blue 
bars according to neuron type) of the network shown in (a) and (b) perfectly agrees 
with the predefined pattern of period T = 1.3 (black bars). The pattern includes 
several simultaneous spikes. One neuron, / = 4, is silenced (non-spiking). 

the domain of [/ms (or U{p^) we have 



Ums{x) = + -] 

a 



-— ln(l — —x) 
7 I 



(20) 



when setting b = —7, a = —I/j. This can be directly verified by explicitely 
inverting Uip. To our knowledge, this has not been noticed before but might be 
useful to establish equivalences for dynamical properties of networks of such 
neurons because the response function H contains both, the rise function U 
and its inverse U~^, cf. Eq. (JSj). 



3 Network Design: 

Analytically restricting the set of admissible networks 



In this section, we explain the underlying ideas of how to design a network. 
For the class of systems introduced above, we derive conditions on a network 
under which it exhibits an arbitrary predefined periodic spike pattern. To 
avoid extensively many case distinctions, the following presentation requires 
that between any two subsequent spike times t and t' of a neuron / that neuron 
receives at least one spike in the interval {t, t') fl (t, t-|-9;). This simply ensures 
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that all spike times in a pattern can be modified by the coupling strengths. 

Definition 1 (Admissible Network) Given a predefined spike pattern, we 
call a network that exhibits this pattern as an invariant dynamics an admissible 
network. 

We assume here that all neuron parameters (f/m, ©m) and delay times Tmi 
are given and fixed in a network; the task is to find networks with these 
given features that exhibit a desired spike pattern as an invariant dynamics. 
To design these networks, we choose to vary the coupling strengths Smi- It 
turns out that there is often a family of solutions such that networks with 
very different configurations of the coupling strengths are admissible; below 
we derive analytical restrictions that define the set of all networks exhibiting 
such a pattern. Of course there might be situations, where other parameters, 
such as the delays [I3| are desired to be variable as well (or only). The key 
aspects of the approach presented below can be readily adapted to such design 
tasks. 

The analysis presented here is very general. It covers arbitrarily large networks, 
different types of neurons, heterogeneously distributed delays and thresholds 
(and thus intrinsic neuron frequencies), combinations of inhibitory and sub- 
and supra-threshold excitatory interactions as well as complicated pattern 
dynamics that include degenerate event times, multiple spiking of the same 
neuron within the pattern and silent neurons that never emit a spike. Figure 
[3] illustrates such a general case. 

3.1 Pattern Periodicity imposes restrictions 

Here we provide an indexing method for any given periodic spike pattern. We 
then explain the relations between the periodicity of a spike pattern and the 
possible) periodicity of a trajectory in state space along which an appropriate 
network dynamical system generates that pattern. 

What characterizes a periodic pattern of precisely timed spikes? Let tj', i' G Z, 
be an ordered list of times at which a neuron emits the spike occurring in 
the network, such that tf > ti' if j' > i'. Assume a periodic pattern consists of 
M spikes. Such a pattern is then characterized by its period T, by the times 
ti G [0, T) of spikes i G {1, ...,M} within the first period, and by the indices 
Si G {1, . . . , N} identifying the neuron that sends spike i at ti . If two or more 
neurons in the network simultaneously emit a spike, i.e. ti = tj with i ^ j, 
the above order is not unique and we fix the corresponding indices Si and Sj 
arbitrarily. The periodicity then entails 

ti + nT = ti+nM and = Si+nM, (21) 
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where n e Z and the definition of s was appropriately extended. This imposes 
conditions on the time evolution of the neurons' phases. Suppose a specific 
neuron / fires at K{1) different times tj^. G [0,T), k G {1, ...,K{1)} within the 
first period. For non-degenerate event times this implies 

MtiJ-^i, (22) 

for the neuron's spike times, whereas at any other time i G [0, T), t 7^ t,^ for 
all k, 

<f>i{t~) < Qi, (23) 

to prevent untimely firing. 

Due to the periodicity of the pattern, we can assume without loss of generality 
that the delay times Tmi are smaller than the patterns period T; otherwise, 
we take them modulo T without changing the invariant dynamics such that 

Tmi e [o,r). 

Theorem 2 The periodicity of the phases of all neurons in the network is 
sufficient for the periodicity of the spiking times of each neuron. If there are 
no supra-threshold excitations in the network, the spike pattern has the period 
of the phase dynamics. 

If the phase dynamics is periodic with period T and no supra-threshold excita- 
tions occur, it satisfies in particular ())i{{ti^+nT)~) = 0; and (f)i{{t+nT)~) < 
for ti 7^ tjj.; tif, G [0, T), k E {I, . . . , K{1)}, are the firing times of neuron / in 
the first period. Therefore the sub-pattern of spikes generated by neuron / is 
periodic with period T. Since I is arbitrary, the entire pattern is periodic with 
period T. 

Interestingly, if there are supra-threshold excitations, the sub-pattern of a 
neuron need not have the period T of the phases, as can be seen from a simple, 
albeit constructed example: Consider a neuron I, which is coupled only to itself 
and receives input from itself as well as once per phase period T from only one 
other neuron m. If neuron / receives a supra-threshold input from neuron m at 
time 9, we have (t)i{9^) < Qi and Ui{(f)i{9')) +eim > Ui{Qi). Suppose the delay 
of the coupling from / to / is tu = T, i.e. equal to the period of the phases, 
and the coupling strength en is inhibitory and such that H^''^^_^_^^^{(f)l{9~)) — 0, 
i.e. Ell = —Ui{(f)i{9^)) — Elm < 0. Then the phase of neuron / can be periodic, 
whether or not it receives a spike from itself because (t)i{9) = in each case, 
either due to the reset of neuron / or due to the inhibitory spike received from 
itself. Now, if neuron / sent a spike at time 9, there will be no spike sending at 
9 + T because of the inhibition by its self-interaction. Since the self-interaction 
spike is then missing at time 9 + 2T, a spike will be emitted at that later time 
and so on. So the spike sub-pattern of this neuron (consisting of all those 
spikes in the total pattern that are generated by neuron /) has period 2T, and 
not T. 
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However the spike sub-pattern of any neuron / has to be periodic even if 
it receives supra-threshold input. This can be seen as follows: Due to the 
conventions above, a spike can only be emitted when there is a discontinuity 
in the phase 0/ (after a supra-threshold excitation, the phase is always zero, 
after a simultaneous reception and spiking it is always unequal to 9^) or if the 
neuron receives a supra-threshold input when its phase is 4>i{0~) = 0. Since 
(f)i{t) is piecewise continuous, in every (finite) time interval [t,t + T) there are 
only finitely many discontinuities, as well as only finitely many times with 
= because the phase is monotonous otherwise. Therefore, given a 
certain phase dynamics, spikes can be emitted by the network only at finitely 
many times in any interval [t,t + T). This implies that there are only finitely 
many combinations of spikes which can be emitted by the network within a 
period T of the phases. Thus, after a finite integer multiple of T, the spike 
patterns have to recur. After this has happened, not only the phases but 
(because here we can choose T to be an arbitrary integer multiple of the 
phase period such that Tim < T without loss of generality) also all spikes in 
transit are the same as at some time before. Since at any time the state of the 
network is fixed by the phases and the spikes in transit, the entire dynamics 
must repeat. So, the pattern is periodic with some period nT, n G N. 

Theorem 3 Let S C {1, ...,iV} be the set of neurons that (i) do not receive 
any supra-threshold excitations and (ii) are firing at least once in the pattern. 
Then, the periodicity T of the entire pattern is sufficient for the periodicity of 
the phases 

Ut)=Mt + nT), (24) 
for all neurons I ^ S , all n G Z and all t G [0, T). 

We disprove the opposite: Suppose, for some / G S and some t, 4>i(t) > 
(f)i{t + T). Then this inequality remains true for all future times t. First, it 
remains true during free time evolution. Because the inputs are identical for 
every period and because the Hf\(j)) are strictly monotonically increasing as 
function of 0, it remains true also after arbitrarily many interactions. There- 
fore, denoting the next firing time of neuron / after time t by tj, we conclude 
that 1 = (piitj) > (pi{{tj + T)~), violating the pattern's periodicity. An analo- 
gous argument shows that if (j)i{t) < (piit+T) for some t, the pattern would not 
be periodic either. Therefore, if the pattern is periodic, the phases of neurons 
/ G S are also periodic and the phases have the period of the pattern. 

As direct consequence from Theorems [2] and [3] we note the important special 
case S = {1, A^}. 

Corollary 4 // all neurons in the network receive only subthreshold input 
and are firing at least once in a pattern, periodicity of the entire pattern is 
equivalent to the periodicity of the phase dynamics and the periods are equal. 
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Remark 5 If a neuron that (i) receives one or more supra-threshold inputs or 
(a) is silenced (i.e. has no firing time in the pattern) has non-periodic phase 
dynamics, its spike sub-pattern can still be periodic. 

(i) If a neuron / receives a supra-threshold input, a small initial deviation from 
the periodic phase dynamics that occurs sufficiently briefly before the input, 
will only change the phase (pi of that neuron but not its next spike time as long 
as the input remains supra-threshold. Since the dynamics continues without 
deviations with respect to the periodic phase dynamics, all future events will 
also take place at the predefined times. Thus there are initial conditions such 
that the phase dynamics is not entirely periodic but the spike pattern is. (ii) 
A sufficiently small initial deviation from the periodic phase dynamics that 
occurs at a silenced neuron can decay without making the neuron fire such 
that the spike pattern stays periodic as without the deviation, although the 
phase of the silenced neuron is not periodic. 

For simplicity, we impose in the following that the phase dynamics of all 
neurons, including those neurons that are silent (i.e. never send a spike) and 
those that receive supra-threshold inputs, are periodic with period T. We 
consider (piit) for t G [0, T) with periodic boundary conditions. All times are 
measured modulo T and spike time labels j are reduced to {1,...,M} by 
subtracting a suitable integer multiple of M. 

3.2 Parameterizing all admissible network designs 

In this subsection we are working towards an analytical restriction of the set 
of all admissible networks for a given spike pattern. We provide a method of 
indexing all spike reception times, and of ordering them in time. The input 
coupling strengths are indexed accordingly. Based on this scheme, we derive 
conditions ensuring the sending of a spike at the pre-defined spike times, pe- 
riodicity of the phase dynamics, and quiescence (non-spiking) of the neurons 
between their desired spike times. A main result of the paper. Theorem [3, 
provides a system of restrictions on the coupling strengths, which separate 
into disjoint constraints for the couplings onto each neuron, cf. Remark El 

Let Oi^j := tj + Tis- be the time when neuron / receives the spike labeled j from 
neuron Sj. Then, for inhomogeneous delay distribution the Oi^j might not be 
ordered in j. Therefore, we define a permutation ai : {1, ...,M} — > {1, ...,M} 
of the indices of spikes received by neuron I, such that 

kr-=(^iM3) (25) 

is ordered, i.e. Oi^j > 9i^i if j > i. If multiple spikes are received at one time, ai 
is not unique. This, however, has no consequence for the collective dynamics 



14 



because all the associated spike receptions are treated as one according to ([9]). 



If neuron / receives multiple, say p{l,j) spikes at time 6ij, we only consider 
the lowest of all indices j' with reception time 9ij' = 9ij. If neuron I receives 
spikes at Mi different times, we denote the smallest index of each reception 
time by ...,jMi{i) such that 

Jn{l) ■.= Jn-l{l)+p{l,Jn-l{l)). (26) 

for n e {2, . . . , Ml}. Here = 1. The first set of equal reception times starts 
with index = 1 and contains p{l, 1) spikes. Therefore, the second set of 
equal reception times has first index j2{l) = p{l, 1) + 1 = p{l,ji{l)) + 
contains j2(0) spikes. This way all indices are defined recursively. 

To keep the notation concise, we skip the argument / in the following (where 
it is clear) as the argument or index of some quantity which is itself a further 
index or a subindex, e.g., of 6i or £/. For instance, we abbreviate Oiji(i) by 
6ij. and p{l-,jk{l)) by p{jk) where appropriate. Furthermore, indices denoting 
different spike receptions of neuron / are reduced to {1, ...,M/} by subtracting 
a suitable multiple of M;. We define Pi{i) G {1, ...,M/} (cf. also Fig. [4]) as the 
index of the last reception time for neuron / before its firing time t,, 

Pi{i) := argmin{ti - ^z,,, | G {1, MJ, U - 6^^ > 0}. (27) 

If there are no simultaneous spikes received by neuron / and if there is no spike 
received at the firing time ti itself, Pi{i) is given by 

Pi{t) = argmin{t, - Oij | J G {1, M}}. (28) 

In the following, if two or more reception times are equal, we will select the 
smallest index and restrict the dynamics only once, using Eqns. ((HI), ((91) and 
the definition of above. Only the total action of all spikes received by 
a neuron / at a particular 9ij. will be restricted, by a single condition. We 
therefore define the sum of the coupling strengths of all spikes received by 
neuron / at time 6i , as 

= ^Is^l^j^) + ■■■ + (29) 

Indeed, ai{ji{l) + k), /c G {0, ...,p(Z, jj(/)) — 1}, are the indices of the p{l,ji{l)) 
different spikes received by neuron I at the ith reception time 9ij-, i G {1,...,M/}. 
If neuron / receives all spikes at different times, we have j = Sis^^^y Let 

Am = 9i,n^, - 9i,n (30) 

be the time differences between two successive different reception times, where 
i + 1 has to be reduced to {1, M;} by subtracting a suitable integer multiple 
of Ml. We now rewrite Eqns. fl22ll and (l23l) for neuron / as a set of conditions 
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Figure 4. (color) Restriction of a neuron's dynamics between its firing events, cf. 
([3T]) and ([32]) . In this example, two spikes arrive between the firing times U and 
of neuron /. The solid line indicates one possible time evolution of the phase (pi{t). 
Between the firing times, (j)i{t) may follow any path within the possibly semi-infinite 
polygon (gray shaded; green dashed lines show other possible trajectories). A too 
large phase at Gijp^^-j^-^ contradicts (f32l) and will lead to early firing (dark red dashed 
line). The phase at Oijp^^.-^ is fixed (red dot). Any other phase inconsistent with the 
equality in (ISTI) would lead to a firing time earlier or later than predefined (light red 
dashed lines). 

on the phases (f)i{6ij.) at the different spike reception times 9ij- in terms of 
the firing times tj^. of that neuron and the spike reception times 9ij.,, i' G 
{!,... ,MJ. 

If the given pattern does not imply the reception of a spike precisely at the 
firing time tj^. (together with the firing times and the delays also the reception 
times are fixed), this results in 

MOup,J=Qi-iU,-k,p,J, (31) 

Ukn) - (32) 

where k e {1,...,K{1)} and i E {I, Mi}\{P{ik)\k e {1, K{1)}}. We note 
that, by definition (l27l) . there is no input to neuron I between the spike(s) 
received at 6i , , and the neuron's next firing time ti, . 

The firing time condition ( I3T1) states that the neuron at time (^ijp^,^-, is as far 
away from its threshold 9; as it needs to be in order to exactly evolve there 
freely in the remaining time tj^ — dijp^^i^y The inequalities (l32ll guarantee 
that the neuron does not spike between the firing times determined by the 
predefined pattern: They ensure that neuron / is far enough from its threshold 
at all other spike reception times and is not firing at any time that is not in 
the desired pattern, t ^U^^. 

Above, we had fixed the convention, that if a spike is received by a neuron 
when it is just about to fire, the spike received is processed after the sending 
of the new spike. If we had used the convention that first the received spike is 
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considered, the "<" in inequality ( 1321) would have been replaced by a "<". Here 
equality, (f)i{6i,j,) = &i — Ai^i, means that the neuron approaches the threshold 
at d^j^^^^ i.e. 4>i{^r,ji^J = 0«! but since the received spike is processed first, an 
untimely spike can be prevented by an inhibitory input. 

If there is one or several spikes received precisely at a predefined firing time 
tjj., supra-threshold excitation can be used to realize the pattern. To account 
for this, the firing time condition (ISTl) and the silence condition (l32l) with 
i = Piiik) + 1 have to be replaced by the conditions 

Udi,n^,,)<Qi-{U,-h3nJ^ (33) 
Ui{Utl))+ei,p^,^)+,>Ui{Qi). (34) 

Here, the strict inequality (i33ll prevents untimely spiking (cf. the dark red 
dashed line in Fig. [4|) and guarantees that the neuron does not reach the 
threshold by its intrinsic dynamics. The second, inequality (l34ll . ensures the 
spiking at U^. However, (l34l) is not an inequality on the phases depending 
at the reception times only, but involves the total coupling of the incoming 
spikes. We note that expression (1331 ) with an equal sign, "=", describes the 
case that the neuron spikes without supra-threshold excitation, because due 
to our above convention, the firing is treated before the spike reception. Then, 
inequality (l34l) is obsolete. So Eq. (l3T]l is the appropriate spike time condition 
also if spikes are received by neuron / when it just reaches threshold. Now, there 
are two cases possible (i) the spikes do not cause a supra-threshold excitation 
f//(0) + £:/,p(ij.)+i < UiiQi) from the reset phase of the neuron or (ii) they 
cause a supra-threshold excitation, C/z(0) +e«,p(jj.)+i > Ui{Qi). In the first case, 

MUJ = (piikjPi.,)+i) = ^i?p(,,)+i(0)' inthesecond0i(tiJ = 0K^z,iP(.,)+i) = 0- 
In the first case, the silence condition ( 132! ) with i = P{ik) + 1 applies such 
that this case does not need a special treatment, in the second, we have the 
inequality ei,p(i^)+i > Ui{Qi) instead. 

Specifying conditions on the phases at these ordered and clustered (simul- 
taneous) spike reception times is equivalent to specifying the phases at the 
unordered and unclustered times because 4>i{0i^i) = (piiOi^j) if 6*; j = Oi^j. 

If there are no simultaneous events, the strengths of coupling onto a particu- 
lar neuron Z, sui, Z' G {1, . . . , A^}, are restricted by K(l) nonlinear equations 
and M — K{1) inequalities originating from (1311) and (l32l) . All the coupling 
strengths in the network realizing a given pattern are thus restricted by a sys- 
tem of J2f=i K{1) = M nonlinear equations and J2fLi{M - K{1)) = {N - 1)M 
inequalities. 

Remark 6 The constraints (equations and inequalities) restricting the cou- 
pling strengths of the network (to be consistent with a predefined pattern) sep- 
arate into disjoint constraints for the couplings onto each individual neuron. 
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In the presence of simultaneous events, for each neuron there are M; — K{1) + 
S{1) inequalities originating from (l33l) . (l34l) and (l32l) . (where 5'(/) is the num- 
ber of supra-threshold excitations, not counting the ones where the spike is 
omitted) and K{1) — S{1) equations originating from the spikings described by 
( l3Tll . We see that simultaneous receptions decrease the number of constraints. 
Again, these constraints separate (remark [6l). This property is due to the fact 
that the pattern is fixed; it turns out (see below) that because of this separa- 
tion, it is easier to find a solution for the coupling strengths that satisfy these 
constraints. 

Fig. [4] illustrates the constraints. After a firing of neuron / at time ti where 
its phase is zero, conditions (l3Ti ) and f l32ll impose restrictions on the phases 
at the spike reception times while the time evolution proceeds towards the 
subsequent firing time tk of neuron /. 

If we now compute explicitely the dynamics of neuron I between two successive 
firing times ti and tk and evaluate the dynamics at the times occurring in fl3Tl) 
and (l32|l . we obtain 



^ffp(i)+2(^ffpW+i(^'JpW+i + < 0« 

^l?p(fc)(---^I.PW+2(^i?pW+i(^'jp{.)+i - + ^l,P{i)+l) 

■ ■■ + ^l,P{k)-l) = 6/ - {tk - Oijp^^^) 

(35) 

in the case of no spike reception at time ti and no supra-threshold excitation 
that generates the spike at tk. 

Now we consider the case that there was a spike reception at time ti. If a 
supra-threshold spike generated the spike time ti from a phase (piit^) < Qi 
and the intrinsic dynamics generates the spike at tk, the set of equations and 
inequalities reads 

^l,P(i}+2 , 

(36) 

{tk - 0l,jprk))- 



- ^l,P(i} + l , 

- ^l,Pii)+2 , 



<P(.)(-<P(,..(A^pw+i) • • • + = 0/ - 
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Alternatively, at U, the threshold can be reached by the intrinsic dynamics 
(f)i{t~) = Qi although a spike is arriving. Here we have to consider two different 
cases: (i) f//(0) + ei^p(i)+i < Ui{Qi), i.e. the spike is subthreshold. This is just 
a special case of ((35]) with ^ijp(,)+i - U = 0. (ii) Ui{0) + ei^P(i)+i > Ui{Qi), 
i.e. the spike is supra-threshold. In this case, we fixed the convention that the 
second spike is omitted and the neuron is reset to zero; therefore system ( l36l l 
is supplemented with the condition 

ei,P(i)+i > Ui{Qi) (37) 

on e;^p(j)+i. 

The above equations also cover the case that a spike is received by neuron 
I at the spike time when neuron I already reached 0;, i.e. Oijp^^^_^^ = t^. 
However, also supra-threshold excitation can then also be used to generate the 
spike tfc- Then, if no spike is received at or if a spike is received when the 
threshold is already reached and no supra-threshold excitation takes place, the 
couplings are restricted by (l35l) where the last equation has to be replaced by 
the inequalities 



V £i,p(i)+i^ 



+A;,p(i)+i) . . . + \p(k)-l) <Ql- {tk - dljp(k))^ 



(0 



+ ^kP(i)+l) • • • + ^l,P(k)-l) + \,P{k)) + ^/,P(fc)+l > Ui{<di) 



r(0 



(38) 



If supra-threshold excitation occurred at time ti and supra-threshold input 
generated the spike at t^, the couplings are restricted by (l36l) (possibly com- 
pleted by (l37l)). where the last equation has to be replaced by the inequalities 

■ ■ ■ + ^l,P{k)-l) < ©Z — (4 — ^l,jp(k))^ 

^KM'^P(fc)(--^l?Pw+2(^«,^'w+i) 

. . . + Az,p(fc)_i) + Az,p(fc)) + £«,P(fc)+i > Ui{<S)i). (39) 



We have thus shown: 



Theorem 7 The set of solutions to the systems (fg5lj-(fg^) for all K{1) pairs 
of subsequent firing times {ti, tk), where i = in, k = in+i, n G {1, . . . , K{1)}, 
provides the set of all admissible coupling strengths ew, I' G {1,...,A^}, of 
incoming connections to neuron I. 
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Figure 5. (color online) Two different networks (a), (c) realize the same prede- 
fined pattern ((b), (d) grey lines). The networks consist of six identical leaky in- 
tegrate-and-fire neurons with Im = 1-2, 7m = 1, ©m = 1- The networks are realiza- 
tions of random graphs where each coupling is present with probability p = 0.8; the 
coupling delay is Tmi = 0.125. A small random perturbation is applied at the begin- 
ning of the second period. The network dynamics (spike times relative to the spikes 
of neuron I = 1, color coded for each neuron), found by exact numerical integration 
[35] shows that in network (a) the pattern is stable and thus regained after a few 
periods (b); in network (c) the pattern is unstable and eventually another pattern 
is assumed (d). Reproduced from Ref. |23j . 

Corollary 8 Solutions to systems analogous to ^3^)-^3M) for all neurons I G 
{1, . . . , N} define all coupling strengths of an admissible network. 

Often (l35ll - (l39ll are under-determined systems such that many solutions exist, 
implying that many different networks realize the same predefined pattern, cf. 
Fig. El This is illustrated in more detail in the next section. Roughly speaking, 
in the absence of supra-threshold excitation, the time of each spike of each neu- 
ron provides one "hard" (equality) constraint on the in general N-dimensional 
set of input coupling strengths of that neuron. The silence conditions pro- 
vide "soft" (inequality) constraints, often not lowering the dimensionality of 
the solution space of coupling strengths. Intuitively a hard restriction can be 
understood by considering a simple example: Consider a network of = 3 
neurons. If one neuron m receives two spikes in a fixed time interval in which 
it does not send a spike itself, the coupling strengths of these spikes are ar- 
bitrary as long as their total impact on the neuron's phase 0^ (advancing or 
retarding) is the same, cf. also Fig. [H This provides one, and not two, hard 
restrictions to the set of input coupling strengths to neuron m. 

In the case of leaky integrate-and-fire or Mirollo-Strogatz neurons, a solution 
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of (I35l)-(l39l). if one exists, can be found in a simple way, because the system 
is then reducible to be linear in the coupling strengths or polynomial in its 
exponentials, respectively. 

Remark 9 There are patterns for which the systems ^3^-^3^, with prede- 
fined neuron properties and predefined delay distribution, do not have a solu- 
tion. 

This means that if the delays and neural parameters are specified, no network, 
independent of how the coupling strengths are chosen, exhibits that predefined 
pattern. This can already be observed from a simple example: consider a non- 
degenerate pattern where neuron / sends three successive spikes and between 
each two successive of these spike times there is precisely one spike received, 
each sent by the same neuron m. Then, the coupling strength eim is fixed (by 
the firing time condition to which (l35l) reduces) to ensure the correct time of 
the second spike of neuron / and cannot be modified to ensure the third one. So, 
if the interval between the second and third spike time does not by coincidence 
match the one determined by the input, the pattern will not be realizable by 
any network. Other, more complicated examples follow immediately. 

This implies that certain predefined patterns may not be realizable in any 
network, no matter how its neurons are interconnected. We note that if we 
allow the neural parameters and delay times to vary as well, the system again 
might have a solution. 

3. 3 Explicit analytical parameterization 

In this sub-section, we will show that an entire class of patterns can, under few 
weak requirements always be realized by a (typically multi-dimensional) family 
of networks. This class consists of simple periodic patterns, in which every 
neuron fires exactly once before the pattern repeats. For a simple periodic 
pattern, we label, without loss of generality, the neuron firing at time ti by /, 
i.e. si = I for I E {1, ...,M = N}. Accordingly we have 9i^m = tm + 'Tim- The 
time differences between two successive spike times of the same neuron equal 
the period of the simple periodic pattern. Thus, for each neuron / the reception 
times of spikes from all neurons of the network are guaranteed to lie between 
two successive firings of neuron /. We note again, that due to the periodicity of 
the pattern, we can assume without loss of generality that the delay times are 
smaller than the patterns period; otherwise, we take them modulo T without 
changing the invariant dynamics. In the following, we require that two simple 
criteria are met. 

Criterion 10 For each neuron its self-interaction delay is smaller than its 
free period, i.e. tu < Tq^i for / G {1, . . . , N}. 
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This criterion ensures that the spike time of each neuron can be modified, at 
least by the self-coupling. If, as we assume throughout the manuscript (see 
section [3|), a neuron / firing only once in the period (here at ti) receives at 
least one spike in the interval {ti,ti + 9;) (or, if 9; > T in {ti,ti + T)), this 
criterion is not necessary to hold for Theorem [T2] below: Theorem [T2] holds for 
any presynaptic neuron sending the spike modifying the spike time (Criterion 
fm appropriately modified). 

Criterion 11 The threshold minus a possible lower bound of the phase plus 
the self-interaction delay for each neuron I is larger than the pattern's period, 

Qi-Bi + m > T. 

This second condition is obsolete if there is no lower bound of the phase, as 
e.g. for leaky integrate-and-fire neurons. 

Given these weak constraints, the following statement holds. 

Theorem 12 For simple periodic patterns, if conditions ( f7^) and ( 177]] are 
satisfied, solutions to ^3R) exist and the set of admissible networks contains 
an N{N — 1) dimensional submanifold in the space of coupling strengths. 

This means that all simple periodic patterns are typically realizable by a high- 
dimensional family of networks. 

We first show that one solution exists, then state another Theorem, which 
explicitly shows that the solution space contains an A^(A^ — l)-dimensional 
submanifold. 

We explicitly construct a trivial solution, where only self-interaction is present, 
while all the other coupling strengths are zero. We consider the one neuron 
system consisting of neuron /. Because of (j)i{ti) = and condition (fTOl ) at the 
reception time of the spike from neuron I to itself, (t>i{(ti + tu)~) = tu holds. 
At time ti + tu the neuron's phase is set to (piiti + tu) = Qi — (T — tu) < 
Qi by choosing the coupling strength en = iyJ,'/(^+^,^)(0K(^/ + r//)")). Here, 

H^^^^{(f)) = Ui{il)) — Ui{(f)) is the inverse of HP{(j)) with respect to e, which 
exists for any and in the domain of f//. Indeed, < (pi^iti + tu)^) < 9/ 
is in the domain of Ui as well as (piiti + tu). The latter is true, even if a lower 
bound is present, because (piiti + tu) = Qi — (T — tu) > Bi due to condition 
[TTl Now, since no further spike is received, the condition Eq. (|3T1) for the spike 
sending time is satisfied and the next spiking will take place at ti + T. Since 
there are no further spike receptions there are no silence conditions (l32l) to be 
satisfied. All neurons taken together as a network without couplings between 
different neurons the pattern is invariant. We now set out to parameterize 
the entire nonempty class of solutions realizing the given pattern. Indeed, for 
simple periodic patterns this can be done analytically: 
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Theorem 13 For any simple periodic pattern, the set of all networks satisfy- 
ing the systems lEMSBi) can be explicitly parameterized. 

The parameterization for each neuron / G {1, . . . , A^} is given as follows 

(i) in the case 9ij 7^ for all j G {1, A^}, 

where k G {2, M;— 1} and the neurons' phases (pi{Oij^), i G {1, Mi}\{Pi{l)} 
at the spike reception times are the parameters that are subject to the restric- 
tions (l32l) . These equations also hold with Oijpi^i^^-^ — = if there is a spike 
reception at ti but no supra-threshold excitation. 

(ii) If there is a spike reception at t;, neuron I already reaches threshold due 
to its intrinsic dynamics 4>i(tl) = 6/, and there is supra-threshold excitation 
immediately after the reset, we have 



ei,P(i)+i >Ui{Qi) - Ui{0), 

ei,P(r)+2 =iy^'|^'^.^^^^^^)(^,jp(,)^, - tl), 

^i,p{i)+k =^0'(e-^^.^^^^^j(0«(^«,ip(o+fc-i) + ^i,p{i)+k~i), 
£i,p{i) =H0l-ltt-ei^^^^^^)iMkjPii)-i) + \p{i)-i), (41) 

where k G {3,...,M; — 1}. The parameters are the neurons' phases (pi^Oij.), 
i G {1, Mi}\{Pi{l), Pi{l) + 1} at the spike reception times that are subject to 
the restrictions (l32ll and ei^p{i)+i which is bounded below by e«,p{«)+i > Ui{Qi). 

(iii) If there is a spike reception at 9ij = ti, and the spike at ti is generated by 
supra-threshold excitation: 
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ei,p(i)+i >Ui{Qi) - UiiMkjPw) + ^'.^(O)' (42) 

where k G {3,..., Mi}. Here the parameters are the neurons' phases (f)i{9ijj, 
i G {1, M/}\{Pi(Z) + 1} at the spike reception times that are subject to 
the restrictions ( l32l) . (i33l ) and eip(i)+i, which is not parameterized but only 
bounded below by a function of 4'i{^i,jp(i)) unless we require that the spike 
precisely excites the neuron to the threshold, i.e. the "=" in the last equation 
is valid. 

These relations follow directly from (I35tl39p by inversion and fl31tl33p . 

Since the Si^i are disjoint sums of couplings Sij , the couplings towards neuron I 
can be parameterized using the parameters for 5; ^ and p{l,ji) — 1 independent 
couplings per reception time 6ij.. 

We now demonstrate the second statement of Theorem [I2l 

In case (i) above, the Jacobian of the couplings with respect to the phases can 
be directly seen to have full rank M/ — 1. Therefore, parameterization ( l40l ) gives 
an Ml — 1-dimensional submanifold of the M/-dimensional space of e/^j. Since 
the j are just disjoint sums of couplings eij, an (A^ — l)-dimensional sub- 
manifold of networks realizing the pattern exists in A^-dimensional e/j-space, 
j G {1, . . . ,N}, I fixed. We further know that the trivial solution of uncou- 
pled neurons with self-interaction constructed above is contained in case (i). 
Therefore, the set of parameters subject to the restrictions (i32l ) is nonempty. 
Since it is open, there is an (A^ — l)-dimensional open set parameterizing 
the submanifold. The product of these submanifolds of all couplings is an 
A^(A^ — l)-dimensional submanifold which is contained in the set of solutions. 



3.4 A note on stability 

Is a pattern emerging in a heterogeneous network stable or unstable? We nu- 
merically investigated patterns in a variety of networks and found that in 
general the stability properties of a pattern depend on the details of the net- 
work it is realized in, see Fig. [5] for an illustration. Depending on the network 
architecture, the same pattern can be exponentially stable or unstable, or 
exhibit oscillatory stable or unstable dynamics. 
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For any specific pattern in any specific network, the linear stability properties 
can also be determined analytically, similar to the exact perturbation analyses 
for much simpler dynamics in more homogeneous networks [331134] . More gen- 
erally, in every network of neurons with congenerically curved rise functions 
and with purely inhibitory (or purely excitatory) coupling, a nonlinear stabil- 
ity analysis |21j shows that the possible non-degenerate patterns are either all 
stable or all unstable. For instance, in purely inhibitory networks of neurons 
with rise functions of negative curvature, such as standard leaky integrate- 
and-fire neurons, Eq. (flGl) with 7 > 0, every periodic non-degenerate spike 
pattern, no matter how complicated, is stable. 

If in the pattern, a neuron receives a spike when it was just about to spike and 
the corresponding input coupling strength is not zero, the pattern is super- 
unstable: an arbitrarily small perturbation in the reception time can lead to 
a large change in the dynamics. These cases, however, are very atypically in 
the sense that when randomly drawing the delay times and the spike times 
in a pattern from a smooth distribution the probability of occurrence of any 
simultaneous events, in particular those leading to this super-instability, is 
zero. Simultaneous spikes sent and simultaneous spike received by different 
neurons do not lead to a super-unstable pattern, because the phase dynamics 
depends continuously on perturbations. 



4 Implementing additional requirements: 

Network Design on Predefined Connectivities 

4-1 Can we require further system properties? 

As we have seen above, the systems of equations and inequalities (l35il-(l39ll 
defining the set of admissible networks is often underdetermined. We can then 
require additional properties from the neurons and their interactions. So far 
we assumed that neurons and delays were given but arbitrary, but network 
coupling strengths, and therefore the connectivity, were not restricted. 

Here we provide examples of how to require in advance additional features 
that are controlled by the coupling strengths. A connection from a neuron / 
to m can be absent (requiring the coupling strength Smi = 0), taken to be 
inhibitory (emi < 0) or excitatory (e^z > 0) or to lie within an interval; in 
particular, we can specify inhibitory and excitatory subpopulations. 

Additional features entail additional conditions on the phases at the spike 
reception times which can be exploited for network parameterization, as we 
here demonstrate for simple periodic patterns, where we employ the same 
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conventions as in sub-section I3.3[ 



(i) If the pattern is non-degenerate, exclusion of self-interaction is guaranteed 
by the conditions 

M0i,i) = ni (43) 
if there is no spike-reception in {ti,6i), and 

M^l,l) - 0i(^i,<7(a-l(O-l)) = \a-^{i)-i (44) 

otherwise, typically reducing the dimension of the submanifold of possible 
networks by N. 

(a) Requiring purely inhibitory networks leads to the accessibility conditions 

- MOuJ <Ai,, (46) 

where i e {1, M;}\{P/(/)}. Since 0«(^iTip{o+i) ^ ^iJpw+i ~ the first in- 
equality is equivalent to (f)i{(^i,jp(i)+i) < ^i(^Up^i)+i)- This guarantees ei^P{i)+i = 

= < 0, due to the 



monotonicity of Ui, such that the couplings summing up to e/,p(/)+i can be 
chosen to be inhibitory or zero. Analogously, the second inequality ensures 
(pii^iji) ^ ^K^z7ii)" T^ote that (l45l) also covers the case of spikes received 
at time ti. Since their action is inhibitory, no supra-threshold excitation can 
occur and ([45]) yields (j)i{ti) = 0i(^«jp(,)+i) ^ ^iJpm+i - ^« = 0- 

To parameterize all networks we can therefore successively choose <Pi{^i,jp(i)+„^)i 
m e {1,...,M/ — 1}, starting with m = 1. Inequalities (l45ll and (l46ll hold 
with reversed relations for purely excitatory coupling if no supra-threshold 
excitation occurs. Otherwise, they have to be replaced by 



U0i,p,,,J>k3P,,^.-tu (47) 
Ukn^.) ~ Uk3.) >^i.: (48) 

where i G {1, M;}\{P;(/), P;(/) + 1}. An additional condition at time ti = 
^i,jp{i)+i '^ot necessary, since the condition that the spike has a supra- 
threshold action already ensures the excitatory coupling. In general, purely 
inhibitory realizations can exist if the minimal inter-spike-interval of each sin- 
gle neuron / is larger than the neuron's free period, i.e. 

min {ti^^^-ti^\k G {1,...,K{1)}} > Qi, (49) 
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for all / G {1, A^}, where the index k + 1 has to be reduced to {1, K{1)} 
subtracting a suitable multiple of K{1). If (l49l) is not satisfied, for some k, 
(f)i{t~^^J = Qi is not reachable from (piiti^) = 0. For the same reason, purely 
excitatory realizations can exist if 

max {U^_^^ -Ujke {1, K{1)}} < Qi. (50) 

In the case of simple periodic patterns, for purely inhibitory coupling the 
inequalities (l49l) reduce to T > maxm 6m. If even 

T > max Qrn (51) 

holds, the trivial solution is purely inhibitory with couplings en < 0. Therefore, 
from Theorems [T2l [TSl and the corresponding proof, we conclude that there is a 
submanifold of purely inhibitory networks in the set of solutions. Analogously, 
if 

T<mine^, (52) 

m 

there is a submanifold of purely excitatory networks in the set of solutions. 

4-2 Very different connectivities, yet the same pattern 

Requiring certain connections to be absent is particularly interesting. This 
just enters the restricting conditions (I35II39I) as simple additional equalities 
^mi = specifying that there is no connection from I to m. 

By specifying absent connections we generally also specify which connections 
are present (except in cases where Emi = by coincidence), i.e. the connectivity 
of the network. Though very simple to implement, specifying the absence of 
connections is thus a very powerful tool. 

Remark 14 Absence of each of the N"^ connections Smi, rn,l G {1, . . . , A^}, 
can be pre- specified independently. 

This means that we can typically specify in advance any arbitrary connectiv- 
ity of the network. A particular predefined pattern is of course not always 
realizable in such a network. 

We illustrate this network design with predefined connectivities by a few exam- 
ples. The two small networks of Figure [5] are both networks with pre-specified 
absent links. Here we chose random networks of A^ = 6 neurons where each 
connection is present with probability p = 0.8. The figure displays two dif- 
ferent networks that exhibit the same pattern. One network has been chosen 
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such that the pattern is stable the other such that it is unstable. Interestingly, 
on the one hand the same pattern can be invariant in two different networks 
with similar statistics, on the other hand their stability properties depend on 
the details of the coupling configurations. 

We also considered large networks by predefining exactly the presence or ab- 
sence of each link according to very different degree distributions. We designed 
them, by varying the remaining (non-zero) coupling strengths, such that all 
network examples exhibit the same predefined simple-periodic pattern. Net- 
work design on specific connectivities is of course not restricted to the example 
cases presented here, because the sets of input coupling strengths can be spec- 
ified independently from each other. 

For illustration, we present four large networks of = 1000 neurons realizing 
the same predefined periodic pattern of spikes. For simplicity, we took for all 
networks the in-degree equal to the out-degree for each neuron. A random 
degree sequence was drawn from the given degree distribution (see below) and 
the degrees assigned to the neurons. The networks were then generated using 
a Monte-Carlo method similar to those discussed in Ref. [24] . 

Approximately 50% of the neurons are of integrate-and-fire type, the remain- 
ing are of Mirollo-Strogatz-type. The parameters of the leaky integrate-and- 
fire neurons are randomly chosen within Im G (1.08,2.08), 7^ G (0.5,1.5), 
the parameters of the Mirollo-Strogatz neurons are randomly chosen in 
bm e (0.9, 1.2), then a„ G (l/(e^'"-l)-0.1, l/(e'''"-l)+0.1). The thresholds of 
both neuron types are uniformly distributed within the interval G/ G (0.8, 1.2). 
The delay distribution is heterogeneous, delays are uniformly distributed in 
the interval Tim G (0.1,0.3), l,m e {1, A^}. 

Two network examples (Figs. [6|7ll have random connectivity with different 
exponential degree distributions 

p{k) oc e""*^ (53) 

where k is the neuron degree. The other two networks (Figs. [8|9ll have power- 
law degree distribution, according to 

p{k) oc k-^ (54) 

For both distributions, we fixed a lower bound on the degree kc = 6 such that 
each neuron has k > kc input and output connections. For networks of both 
distributions, we realized one with purely inhibitory coupling strengths (Figs. 
[6|8l) and one with mixed inhibitory and excitatory coupling strengths (Figs. 

All network examples are constructed to realize the same predefined spike 
pattern with period T = 1.5. The numerical simulations (Figs. [6H9t, green 
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or blue bars for spiking integrate-and-fire or Mirollo-Strogatz-type neurons) 
agree perfectly with the predefined pattern (Figs.[6H9t, underlying black bars). 



Remark 15 Due to the simplicity of imposing absence of links, the same 
method can be applied to a wide variety of network connectivities. In particular, 
a connectivity can be randomly drawn from any kind of degree distribution; a 
connectivity can also be structured (e.g. correlated degrees) and one may want 
to implement a very detailed specific form of it, e.g., as given by real data. 

As noted above, however, not all networks can be designed for any pattern; in 
particular it is in general necessary to have sufficiently many incoming links 
to each neuron such that the interaction delay times and the input coupling 
strengths can account for the desired phase dynamics consistent with the 
predefined spike pattern. 




Figure 6. (color) Network design with given connectivity. Predefined pattern in a 
network {N = 1000) with exponential degree distribution (panel (a), a = 0.03) and 
purely inhibitory coupling. Panel (b) displays the sub-matrix of coupling strengths 
between the first 50 neurons. Inhibitory couplings are red, excitatory couplings are 
gray. The intensity of the color is proportional to the coupling strength. Due to 
too faint color, some very weak couplings are invisible in the plot. The frame shows 
integrate-and-fire neurons in green and Mirollo-Strogatz neurons in blue, (c) The nu- 
merical simulations of the designed networks (green and blue bars for integrate-and- 
fire and Mirollo-Strogatz type neurons) show perfect agreement with the predefined 
pattern (black bars). 
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Figure 7. (color) Network design with given connectivity. Predefined pattern in a 
network {N = 1000) with exponential degree distribution (panel (a), a = 0.1) and 
mixed inhibitory and excitatory coupling. Other panels as in Figure [H 
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Figure 8. (color) Network design with given connectivity. Predefined pattern in a 
network {N = 1000) with power-law degree distribution (panel (a), 7 = 3.0) and 
purely inhibitory coupling. Other panels as in Figure [6l 
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Figure 9. (color) Network design with given connectivity. Predefined pattern in a 
network {N = 1000) with power-law degree distribution (panel (a), 7 = 2.5) and 
mixed inhibitory and excitatory coupling. Other panels as in Figure [H 

5 Designing optimal networks 

In section [3] we derived analytical constraints specifying the set of all net- 
works that exhibit a predefined pattern and found that often there is a multi- 
dimensional family of solutions in the space of networks (as defined by all 
coupling strengths). In the previous section we exploited this freedom to de- 
sign networks the connectivity of which is specified in detail. We may also 
exploit the freedom of choosing a solution among many possibilities by opti- 
mizing certain network properties. 

Can we design networks that optimize certain structural features and at the 
same time exhibit a predefined pattern dynamics? This question is a very 
general one and it can be addressed by considering a variety of features of 
neuroscientific or mathematical interest. To briefiy illustrate the idea, we here 
focus on optimizing convex 'cost' functions of the coupling strengths eim and 
look for those networks among the admissible ones that minimize wiring costs. 

Even for this very specific problem there are a number of different approaches 
we can take. For instance, we can consider networks with the same type of in- 
teractions, inhibitory or excitatory, or allow for a mixture of both, or optimize 
for different features of the connectivity. For simplicity, we here consider small 
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Figure 10. (color) Network of leaky integrate-and-fire neurons that minimizes the 
wiring cost in Euclidean norm by minimizing (l55l) . The parameters are randomly 
chosen within G (1.0,2.0), 7^ G (0.5,1.5) and 6^ G (0.8,1.2). The delays are 
uniformly distributed in G (0.1,0.9), l,m € {1,...,A^ = 16}. Panels (a) and 

(c) show the network and the coupling matrix eim- Panel (b) shows the histogram 
of the strengths of existing connections in the network. The bin size is 0.05. Panel 

(d) displays the predefined spike pattern (black bars) that is accurately reproduced 
(green bars). In the optimal network every neuron is connected to every other ex- 
cept the silenced neuron I = 4. This neuron has no outgoing connections: Since it 
generates no spikes, outgoing connections would be superfluous and do not appear 
in the optimal network. 

networks whose neurons are exclusively of integrate-and-fire type and allow 
for a mixture of inhibitory and excitatory coupling. Integrate-and-fire neurons 
have the advantage (for both analysis and optimization) that the constraints 
(I35D-(I39D are linear. 

The most straightforward goal for optimizing wiring costs is to minimize the 
quadratic cost function 

N N 

G{e):=j:T.^L, (55) 

1=1 m=l 

A similar approach has already been successfully used when minimizing wiring 
costs of biological neural networks based on anatomical and physical con- 
straints but neglecting dynamics issues, see, e.g. [Sj. When minimizing the Eu- 
clidian (L2) norm ^JG{e) by minimizing (l55l) for each row vector {ei^rn)me{i,...,N} 
of the coupling matrix, a solution is searched among the admissible ones that 
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Figure 11. (color) Network that minimizes the wiring cost in Li-norm (|56p . 
The parameters are randomly chosen within Im £ (1.0,2.0), 7^ € (0.5,1.5) 
and Qm ^ (0.8,1.2). The delays are uniformly distributed in Tim ^ (0.1,0.9), 
l,m € {1,...,A^ = 16}. Panels (a) and (c) show the network and the coupling 
matrix eim- Panel (b) shows the histogram of the strengths of existing connections 
in the network. The bin size is 0.05. Panel (d) displays the predefined spike patterns 
(black bars) that is accurately reproduced (green bars). The optimal network is very 
sparsely connected. In fact the network has one large strongly connected compo- 
nent, containing the neurons {1, 2, 3, 5, 7, 9, 10, 13, 14}, while the remaining neurons 
receive connections exclusively from this component and do not have any outgoing 
connections. 



is closest to the origin in the space of networks (defined by the coupling 
strengths). 

Figure [10 shows an example of such an optimization. The network is almost 
globally connected and shows moderate variation among the individual cou- 
pling strengths. The predefined pattern dynamics is exactly reproduced. Such 
a network, while optimizing the wiring cost according to (l55ll does not appear 
to have any special features apart from apparently homogeneous and relatively 
small coupling strengths. 

It seems that nature often designs networks in a different way, possibly such 
that they serve a dynamical purpose especially well. In particular evolution 
has not optimized most biological neural networks in the above manner: they 
are not close to globally coupled. 
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An alternative goal for optimizing wiring costs is to minimize the cost function 

N N 

G{s):=J2E\^irn\, (56) 

1=1 m=l 

that is, the Li-norm of each row vector of the coupling matrix. When minimiz- 
ing the Li-norm (i56l l. as before, a solution is searched among the admissible 
ones that is closest to the origin in the space of networks, but this time 'close' 
is defined by the Li distance measure. Interestingly, under weak conditions 
on the linear equality constraints, an optimal solution (l56l) . searched under 
these constraints only, has many entries eim equal to zero, cf. [7|. Because we 
typically also have many inequalities which depend on details of the pattern 
dynamics and are therefore uncontrolled, we cannot guarantee the zero entries 
for the full optimization problem (defined by equalities and inequalities) here. 
However, our numerics suggests that the solution in fact gives a network with 
many links absent and the number of links present being typically of the order 
of number of equality constraints. 

Thus a network optimized by minimizing the Li-norm is sparse, see, e.g.. Fig. 
[TTl Moreover, compared to the optimal L2-norm solution above, this network 
has more heterogeneous connection strengths. Given some type of dynamics, 
a sparse network possibly is what biological systems would optimize for. In 
biological neural networks for instance, creating an additional synapse would 
probably use more resources (energy, biological matter, space, time, etc.) than 
making an existing synapse stronger. 

Sparseness might possibly also be optimized in biological neural networks 
where requirements are met enabling other specific, functionally relevant dy- 
namics. In general, of course, this dynamics may or may not consist of spike 
patterns. 

Remark 16 The optimization problem, and ([5^ with constraints ^SE)- 
(E^, does typically not have a true optimum. 

If a pattern is predefined that has more than one reception times between two 
successive sending events of some neuron, there usually are strict inequalities 
among the constraints (l35l ) - ([39l) . Because the functions in (l35l) - fl39ll are 
local homeomorphisms (i.e. are continuous with local inverses that are con- 
tinuous) the set of admissible coupling strengths is then not closed and thus 
does not contain its boundary. 

During optimization, typically a solution is searched that is as close to such 
a boundary as possible. For instance, suppose one connection from m to / is 
inhibitory and its strength eim is desired as small as possible. Then a solution 
is searched where the phase (pi of the neuron I that receives a spike from m 
is such that the phase jump that spike induces is maximal (in absolute value) 
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when Elm is held constant. This way a given desired phase jump would be 
achieved by a minimal coupling strength. Typically, the phase (f)i sought-after 
corresponds to a boundary of the set of admissible phases. For instance, if Ui is 
concave, an inhibitory spike has the largest possible effect on 0; (largest phase 
jump) at 0i = 9;. The corresponding phase constraint, however, may read 
(pi < Qi. Thus the boundary phase and therefore also the boundary coupling 
strength cannot be assumed. As a consequence, the optimization problem has 
no true solution. 

We fix this problem by imposing, instead of (l35l) - (l39l) and possible additional 
constraints with inequalities of the type 0; > x or 0; < y, constraint sets that 
are closed, i.e. (pi > x + k or (pi < y — where k > 0, k <^ 1 is a small cutoff. 
We fixed k = 0.001 in the optimal design problems considered here. 



6 Brief Network Design Manual 

In this section we briefly summarize the presented method (of designing the 
coupling strengths of a network such that it realizes a pre-defined pattern) 
by providing step-by-step instructions. For simplicity, as above, we assume 
that all other parameters, such as neuron rise functions and interaction delay 
times are given or fixed a priori. We refer to the relevant sections and formulas 
derived above where appropriate. A simple example of a small network of = 
3 neurons (Fig. [T2|) illustrates the indexing used in the general instructions. 

Suppose a periodic pattern of M spikes is given in a network of A^ neurons. 

1) Label the neurons arbitrarily by m G {1, . . . , A^}. 

2) Fix the origin of time, t = 0, arbitrarily and pick an interval of length T, 
the period of the given pattern. 

3) Order the spike times. Some neurons may send one spike per period, others 
multiple spikes, and again others no spike at all (silent neuron). Label the times 
of all spike sending events according to their temporal order of occurrence in 
the network. In the example of Fig. [121 we have one spike time ti of neuron 
m = 3, two spike times t2 and of neuron m = 2 and one spike time of 
neuron m = 1. 

4) Compute the spike reception times at each neuron / using the interaction 
delay times Tim such that 6ij = tj + Tim- Here m is that neuron that sent the 
spike at time tj. We identify this neuron by Sj := m in the formulas above. For 
those neurons I for which the spike reception times are not ordered, reorder 
them by permuting indices according to (l25l) to obtain ordered reception times 
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Figure 12. Pattern of M = 4 spikes in a network of = 3 neurons illustrating the 
indexing of spike sending and reception times. The spike (sending) times U, marked 
by black bars, are indexed with increasing i according to their temporal order of 
occurrence in the network (the neuron identities play no role for this index). The 
ordered spike reception times 0; j are displayed for neuron 1 = 2. They are generally 
different for other receiving neurons (l 7^ 2, not shown) and obtained by adding 
the delay times r/^ (dashed lines) to the spike sending times tj and then ordering 
the resulting set for each neuron. Here there is one degenerate event: neuron I = 2 
receives a spike from m = 1 exactly at its second spike sending time (light gray 
vertical bar). 

61 J. In the example, the delay time T23 from neuron m = 3 to neuron / = 2, 
is longer than T22, which, for the given pattern, results in reception times 62 j 
that are not in the same order as the spike sending times tj. Particularly we 
have 02,1 = 6*2,2 , ^2,2 = 6^2,1, ^2,3 = 6*2,3 and ^2,4 = 6*2,4 . The ordered reception 
times 62J are as indicated in Figure fT2l 

5) Are there degenerate times at which a reception time at one neuron equals 
that neuron's spike sending time? If so, decide whether to use, for each such re- 
ception, supra-threshold or sub-threshold input signals; for each non-degenerate 
spike reception, use sub-threshold inputs. In the example, the time at which 
neuron 2 receives a spike from neuron 1 coincides with the second spike send- 
ing time ^4 = ^2,3 of neuron 2. So for this reception time ^2,3 of neuron 1 = 2, 
decide whether to use sub- or supra-threshold input. For all other receptions 
at neuron 1 = 2, use sub-threshold input. 

6) For each neuron / and each spike time of that neuron, look for the 
previous spike time of neuron / and name it "t". Compute and look up the 
particular response functions H^, the thresholds 6; and the differences in 
spike reception times Aij. Now, if there is 

(a) no spike reception at time and no supra-threshold input generating tk 
write down system ( !35l) . 

(b) a spike reception at ti inducing the spike at ti by a supra-threshold input 
and no supra-threshold input generating tk, write down system fl36ll . 

(c) a spike reception at time ti but the threshold is nevertheless reached by 
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the neuron from its intrinsic dynamics (as desired by the designer) and no 
supra-threshold input generating tk : if the coupling, effective after reset at 
ti, is (i) subthreshold, this is a special case of fl35ll : (ii) if it is supra-threshold, 
supplement (l36l) with (l37l) . 

(d) case (a) with supra-threshold input generating tk write down f l35ll with the 
equation replaced by ( l38l) . 

(e) case (b) with supra-threshold input generating tk write down (l36D and re- 
place the equation by (l39l) . 

(f) (i) for the case (c,i) with supra-threshold input generating tk, write down 
( l35l) and replace the equation by (l38ll (ii) for the case (c,ii) write down ( l36l ) 
completed by (1371 ) and replace the equation by (1391 ). 

Repeat this step 6) for all neurons / and all pairs (ti,tfc) of their successive 
spike times. 

At this point, a complete list of restricting equations and inequalities has been 
created. One particular solution to these restrictions provides all coupling 
strengths of a network that exhibits the predefined pattern as an invariant 
dynamics. The set of all solutions thus provides the set of all networks that 
exhibit this spike pattern. 

One can now either 

7) solve for one particular solution; or 

8) further restrict the constraint system, e.g. by requiring additional properties 
of the connectivity, cf. section HJ and solve that for a particular solution; or 

9) use the entire constraint system and try to find a solution that is optimal in 
a desired sense, as done in section [5] for the example of minimal wiring costs; 
or 

10) combine additional restrictions, point 8), and optimization, point 9). 

Point 10) has not been presented in this manuscript but is an interesting 
starting point for future research. 

We found it useful to start trying these network design methods on small 
network examples of simple units, for instance integrate-and-fire neurons, and 
investigate very simple patterns with few (or no) degeneracies first. Moreover, 
given that there is no general recipe about how to apply additonal restrictions 
and how to solve general optimization problems, it might also be useful to start 
with few restrictions and simple optimization tasks in very small networks 
the dynamics of which (and possibly their desired "optimal" features) can be 
understood intuitively. 
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7 Conclusions 



7. 1 Summary 



In this article, we have shown how to design model networks of spiking neurons 
such that they exhibit a predefined dynamics. We focused on the question of 
how to adapt the coupling strengths in the network to fix the dynamics. We 
derived analytical constraints on the coupling strengths (which define the set 
of all networks) given an arbitrarily chosen predefined periodic spike pattern. 
The analysis presented here is very general. It covers networks of arbitrary 
size and of different types of neurons, heterogeneously distributed delays and 
thresholds (and thus intrinsic neuron frequencies), combinations of inhibitory 
and sub- and supra-threshold excitatory interactions as well as complicated 
stored patterns that include degenerate event times, multiple spiking of the 
same neuron within the pattern and silent neurons that never fire. These 
constraints do not admit a solution for certain patterns. Once the features 
of individual neurons and the delay-distribution are fixed, this implies that 
these patterns cannot exist in any network, no matter how the neurons are 
interconnected. 

A predefined simple periodic pattern is particularly interesting because under 
weak assumptions, the constraint system has a solution for any such pattern. 
Thus, a network realizing any simple periodic pattern is typically guaranteed 
to exist; we analytically parameterized all such networks. The family of solu- 
tions is typically high-dimensional, cf. also [38], and we showed how to design 
networks that are further constraint. We highlighted the possibility to design 
networks of completely predetermined connectivity (fixing the absence or pres- 
ence of links between each pair of neurons). To illustrate the idea, we have 
explicitely designed networks with different exponential and power-law degree 
distributions such that they exhibit the same spike pattern. 

The design perspective can furthermore be used to find networks that exhibit 
a predefined dynamics and are at the same time optimized in some way. As 
a first example, we considered networks minimizing wiring cost. The connec- 
tivity of biological neural networks that exhibit precise spatio-temporal spik- 
ing dynamics is typically sparse. The work presented here suggests that this 
sparseness may result from an optimization process that takes into account 
dynamical aspects. If biological neural networks indeed optimize connectivity 
for dynamical purposes, our results suggest that these networks may minimize 
the total number of connections (rather than, e.g., their total strengths) and 
at the same time still realize specific spiking dynamics. 
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1.2, Perspectives j or juture research 



The dynamics of artifically grown biological neural networks may provide an 
immediate application ground for the theory presented here. For instance, 
to uncover the origin of recurring, specific spike patterns, one could imagine 
using a design approach to precisely control the growth of biological neural 
networks on artificial substrates and reveal under which conditions and how 
a desired pattern arises in a biological environment. For practicability of such 
an approach, of course, pattern stability, only briefiy discussed here, needs a 
more detailed analysis. Moreover, the size of the basin of attraction of a spike 
pattern will probably also play an important role in such studies. Perhaps it 
may even become possible to develop design techniques to optimize pattern 
stability and basin size, thus gaining robust pattern dynamics. 

Network design might be a valuable new perspective of research, as shown here 
by example for spiking neural networks. Using the design idea might not only 
aid a better understanding of the relations between structure and function 
of complex networks in general; network design might also be exploited for 
systems that we would like to fulfill a certain task, for example computational 
systems such as artificial neural networks. 

The idea of designing a system of coupled units is not new. For instance an 
artificial Hopfield neural network [16] can be trained by gradually adapting 
the coupling strengths, such that it becomes an associative memory, fulfilling 
a certain pattern recognition task. Such networks typically consist of binary 
units that are all-to-all coupled. However, already in the late 1980's [6j mean 
field theory has been successfully extended to study the properties of sparse, 
randomly diluted Hopfield networks. In that work, Derrida, Gardner and Zip- 
pelius showed that the storage capacity of such diluted systems is reduced 
compared to the all-to-all coupled one, but still significant. 

Here we transferred the idea of system design to complex networks that may 
have a complicated, irregular connectivity and thus cannot in general be de- 
scribed by mean field theory. In related study [39], a method has been pre- 
sented to construct neural network models that exhibit spike trains with high 
statistical correlation to given extracellular recordings. The specific results 
presented our this study might be valuable to obtain further insights into bio- 
logical neural systems and the precisely timed, still unexplained, spike patterns 
they exhibit. This study, however, also raises a number of questions both for 
the theory of spiking neural network as well as, more generally, for studies of 
other complex networks and their dynamics. We list a few questions we believe 
are among the most interesting, and promising in the near future: 

Can network design studies help to develop functionally relevant dynamics? 
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Design of particular model networks could on the one hand identify possible 
functional (as well as irrelevant) subgroups of real-world networks, including 
neural, gene and social interaction networks; on the other hand network design 
could also guide the development of new useful paradigms and devices, for 
instance for information processing or communication networks. 

What is an optimal network design that ensures synchronization [28], a promi- 
nent kind of collective dynamics? The approach could of course also be useful 
to avoid certain behavior. For instance, may network design even give hints 
about how to suppress synchronization and hinder epileptic seizures in the 
brain (see e.g. [27] and references therein)? What are potential ways to de- 
sign your favorite network? What kind of dynamics would be desirable (or 
undesirable*) for it. 

Let's use network design - and make specific network dynamics (not*) happen. 
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